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Abstract 

Version  1.0  of  the  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  was  released  in  September  2012.  The  SEBoK 
contained  over  1000  articles  and  almost  700  books  as  references.  224  of  the  references  were  cited  as  primary  references.  This 
paper  disaggregates  the  references  by  several  dimensions.  Interesting  insights  include  the  most  cited  references;  topics  where 
references  are  sparse,  indicating  immaturity  of  the  field;  the  longitudinal  distribution  of  references;  organizations  with  the  most 
authors  cited;  and  the  most  active  research  topics. 
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1.  Introduction 

The  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  [1]  summarizes  the  literature  about  systems 
engineering.  This  paper  describes  the  SEBoK,  and  analyzes  the  references  cited  in  the  SEBoK.  Section  one 
describes  the  genesis  of  the  SEBoK.  Section  two  describes  the  methods  employed  in  this  paper,  and  some  caveats. 
Section  three  provides  a  Pareto  description  of  the  primary  references.  Section  four  discusses  topics  with  the  most 
and  fewest  references.  Section  five  presents  a  longitudinal  analysis  of  aspects  of  the  SEBoK.  Section  six  presents 
the  affiliation  of  authors  cited  by  the  SEBoK.  Section  seven  presents  some  initial  usage  data  on  the  SEBoK. 
Section  eight  concludes  with  some  comments. 

1.1.  BKCASE  Project 

In  September  2009,  the  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering  (BKCASE™) 
project  was  established  with  primary  sponsorship  by  the  U.S.  Department  of  Defense  (DoD)  through  the  Systems 
Engineering  Research  Center  (SERC).  It  was  a  university-led  partnership  between  Stevens  Institute  of  Technology, 
Hoboken,  NJ  and  Naval  Postgraduate  School,  Monterey,  CA,  with  support  from  various  other  universities, 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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organizations,  government  agencies,  and  professional  societies.  The  objective  of  the  BKCASE  project  was  to  create 
and  deliver  two  products:  (1)  the  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  by  September 
2012  and  (2)  the  Graduate  Reference  Curriculum  for  Systems  Engineering  (GRCSE)  by  December  2012.  Both  these 
products  are  intended  for  worldwide  use,  available  for  public  use  at  no  cost.  At  the  time  of  writing  this  paper, 
version  1.0  of  SEBoK  had  been  released  on  September  14,  2012  [1]  and  version  1.0  of  GRCSE  was  on  track  for 
release  on  December  15,  2012. 

1.2.  The  SEBoK 

The  SEBoK  is  a  major,  peer-reviewed  addition  to  the  systems  engineering  literature.  It  consists  of  7  parts 
broken  into  26  knowledge  areas,  with  113  full  length  articles  on  topics.  There  are  5  use  cases,  7  case  studies,  and  6 
vignettes  to  illustrate  the  contents.  The  glossary  has  364  entries,  and  there  are  224  primary  references.  The  full  set 
of  references  includes  over  1000  articles  and  700  books,  written  by  over  2000  different  authors.  It  is  over  850  pages 
long.  The  project  took  three  years. 

The  Office  of  the  Secretary  of  Defense  was  the  primary  sponsor.  Partners  in  the  project  included  the 
International  Council  on  Systems  Engineering,  the  Institute  of  Electrical  and  Electronic  Engineer  Computer  Society, 
the  Institute  of  Electrical  and  Electronic  Engineer  Systems  Council,  the  Association  for  Computing  Machinery,  the 
Institute  for  Industrial  Engineers,  and  the  National  Defense  Industrial  Association. 

The  SEBoK  author  team  includes  70  contributors  from  six  continents.  Comments  from  several  hundred 
reviewers  were  submitted,  and  each  comment  was  addressed,  logged,  and  the  response  published. 

The  SEBoK  was  published  in  a  Wiki  format  for  ease  of  updating  and  for  community  participation  in  its 
maintenance. 

2.  Methods 

Students  supporting  the  primary  author  constructed  a  database  in  the  summer  of  2012  of  every  reference  in  the 
SEBoK.  Queries  of  that  database  provide  some  of  the  data  in  this  paper.  Other  data  was  assembled  by  use  of  the 
organic  search  capabilities  in  the  Wiki  hosting  the  SEBoK. 

A  few  references  were  added  during  final  editing  of  the  SEBoK,  and  after  the  construction  of  the  database. 
Accordingly,  some  of  the  data  here  may  be  slightly  inexact.  The  authors  claim  that  this  possible  imprecision  does 
not  detract  from  the  main  points  below. 

3.  Primary  references 

Some  224  of  the  references  in  the  SEBoK  were  designated  as  “Primary  References.”  That  meant  a  reference  was 
key  to  understanding  one  or  more  topics,  and  was  recommended  for  reading.  Some  of  the  primary  references  were 
more  heavily  cited  than  others.  Table  1  gives  the  number  of  topics  in  the  SEBoK  that  list  a  given  primary  reference, 
in  descending  order,  for  those  cited  in  six  or  more  topics. 

Table  1 :  Number  of  Topics  citing  a  Primary  Reference. 


Primary  Reference 

Number  of  topics 
citing 

INCOSE  Handbook  [2] 

40 

ISO  15288  [3] 

25 

Systems  Engineering  and  Analysis  [4] 

15 

A  Journey  Through  the  Systems  Landscape  [5] 

12 

Systems  Theory,  Systems  Practice  [6] 

12 

Managing  and  Leading  Software  Projects  [7] 

11 

NASA  Systems  Engineering  Handbook  [8] 

11 
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General  Systems  Theory:  Foundations,  Development,  Applications  [9]  1 1 

A  Guide  to  the  Project  Management  Body  of  Knowledge  [10]  10 

Visualizing  Project  Management  [11]  9 

Defense  Acquisition  Guidebook  [12]  9 

Software  Engineering  Body  of  Knowledge  [13]  9 

A  Case  for  Service  Systems  Engineering  [14]  7 

ANSI/EAC  632  [15]  7 

INCOSE  SE  Vision  2020  [16]  6 

Essentials  of  Project  and  Systems  Engineering  Management  [17]  6 

Service  Systems  Engineering  and  Management  [18]  6 


Table  one  indicates  that  the  INCOSE  Handbook  and  1SO/IEC  15288  are  the  most  heavily  internally  cited  SEBoK 
primary  references,  which  is  not  surprising.  The  text  by  Blanchard  and  Fabrycky  [4]  is  third.  Four  of  the  top 
seventeen  references  are  on  project  management.  Five  are  standards  or  handbooks.  Three  are  works  on  systems 
science.  Sixteen  of  the  seventeen  are  books;  only  one  of  the  top  primary  references  is  an  article.  This  paucity  of 
journal  articles  likely  reflects  the  definition  of  primary  reference. 

4.  Topics  and  references 

Each  of  the  113  topics  had  a  number  of  references.  Not  counting  Part  1,  which  was  primarily  administrative,  and 
part  7,  which  was  implementation  examples,  the  authors  examined  the  distribution  of  the  number  of  citations  per 
article.  Parts  2  through  5  average  about  20  references  per  article;  part  6  has  about  10  references  per  article.  Figure  1 
shows  the  distribution  of  counts. 


Figure  1 :  Boxplot  of  the  number  of  references  in  each  topic,  by  Part.  Part  six  has  significantly  fewer  references  than  the  other  parts. 

Articles  with  relatively  fewer  references  include: 

•  System  fundamentals  (5) 

•  Why  model  (4) 

•  Analysing  and  selecting  a  solution  (4) 

•  Implementing  and  developing  a  system  solution  (5) 

•  Deploying,  using  and  sustaining  systems  to  solve  problems  (5) 

•  Life  cycle  characteristics  (4) 
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•  System  deployment  (6) 

•  Operation  of  a  System  (6) 

•  Systems  maintenance  (6) 

•  Information  management  (6) 

•  Capability  engineering  (4) 

•  The  nature  of  software  (7) 

•  Systems  engineering  and  project  management  (3) 

•  Relationships  between  systems  engineering  and  project  management  (5) 

•  The  influence  of  project  structure  and  governance  on  systems  engineering  and  project  management 
relationships  (4) 

•  Integration  of  specialty  engineering  (2) 

•  System  assurance  (6) 

•  Manufacturability  and  producibility  (5) 

Fewer  references  can  indicate  immaturity  of  the  topic  area,  although  in  some  cases  it  can  reflect  variability 
between  the  authors  of  the  topics.  The  data  indicates  that  the  literature  on  the  interface  of  systems  engineering  with 
other  supporting  disciplines  is  thin. 

Subsequent  revisions  of  the  SEBoK  will  concentrate  on  expanding  the  reference  sets  for  those  topics  that  currently 
have  few  references.  Community  recommendations  for  references  are  accepted  on  each  topic  page  by  use  of  the 
D1SQUS  comment  feature. 

Some  topics  had  a  relatively  richer  set  of  references.  They  include: 

•  Patterns  of  systems  thinking  (52) 

•  System  life  cycle  process  models:  Vee  (50) 

•  System  life  cycle  process  models:  Iterative  (72) 

•  Service  life  extension  (60) 

The  more  heavily  referenced  topics  are  likely  partially  due  to  the  enthusiasm  of  the  topic  authors. 


5.  Some  longitudinal  data 

The  primary  references  that  were  books  were  sorted  by  the  year  written,  and  then  plotted  in  Figure  2,  which 
shows  that  most  of  the  primary  references  texts  were  written  in  the  last  ten  years. 


♦  Primary  Refs 


Figure  2:  Primary  references  by  year  written. 


This  is  consistent  with  the  emergence  of  systems  engineering  as  a  discipline  in  the  last  two  decades. 


6.  Affiliations 
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Over  2000  authors  are  cited  in  the  SEBoK.  The  affiliation  of  about  80%  of  the  authors  is  known;  some  sources 
do  not  list  the  affiliation  of  their  authros.  Table  2  shows  institutions  with  ten  or  more  authors,  where  known,  cited  in 
the  SEBoK.  This  table  gives  an  insight  into  the  impact  of  those  institutions  on  the  field  of  systems  engineering. 

Table  2:  Number  of  authors  affiliated  with  institutions 


Institution 

Number  of  authors 

Massachusetts  Institute  of  Technology 

50 

Carnegie  Mellon  University 

45 

Stevens  Institute  of  Technology 

38 

Harvard  University 

25 

University  of  Southern  California 

20 

California  Institute  of  Technology 

19 

Naval  Postgraduate  School 

17 

MITRE 

17 

IBM 

16 

Air  Force  Institute  of  Technology 

15 

University  of  Michigan 

15 

Idaho  National  Laboratory 

14 

Loughborough  University 

13 

UC  Berkeley 

12 

Keio  University 

11 

University  of  Virginia 

11 

Georgia  Institute  of  Technology 

10 

George  Mason  University 

10 

7.  SEBoK  usage 

Patterns  of  viewing  for  the  SEBoK  are  still  immature  as  of  the  writing  of  this  article,  since  the  Wiki  has  only  been 
posted  for  3  weeks.  However,  some  trends  are  apparent.  The  main  splash  page  has  been  viewed  the  most,  by  an 
order  of  magnitude.  The  next  most  popular  page  is  the  download  page,  which  has  been  viewed  2700  times.  If 
readers  only  use  the  PDF  version,  their  usage  statistics  will  not  be  recovered. 

Table  3  presents  the  page  views  of  the  administrative  and  content  pages  of  the  SEBoK.  These  give  an  insight 
into  the  impact  of  the  various  topics  of  the  SEBoK,  and  where  the  community  is  seeking  the  most  information. 


Table  3:  Viewing  data  for  the  most  viewed  pages  of  the  SEBoK. 


Page  name 
Main  splash  page 
Download  the  SEBoK 
Systems 

SEBoK  1 .0  Introduction 

Systems  Engineering  and  Management 

Table  of  contents 

Acknowledgements 

Applications  of  Systems  Engineering 


Views  (thousands)  Admin  or  content 


20.1 

Admin 

2.7 

Admin 

1.9 

Content 

1.7 

Admin 

1.3 

Content 

1.0 

Admin 

1.0 

Admin 

.8 

Content 
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How  to  read  the  SEBoK 

.8 

Admin 

Enabling  Systems  Engineering 

.7 

Content 

Life  Cycle  Models 

.7 

Content 

Systems  Engineering  Implementation 
Examples 

.6 

Content 

Systems  thinking 

.6 

Content 

Note  to  Community 

.6 

Admin 

Related  disciplines 

.6 

Content 

Scope  of  the  SEBoK 

.6 

Admin 

Enterprise  Systems  Engineering 

.6 

Content 

What  is  a  System 

.5 

Content 

Systems  Engineering  Overview 

.5 

Content 

System  Fundamentals 

.5 

Content 

System  Definition 

.5 

Content 

Systems  and  Software  Engineering 

.5 

Content 

Representing  Systems  with  Models 

.5 

Content 

The  topic  with  the  greatest  amount  of  visits  is  the  one  on  “Systems.”  That  was  a  surprise  to  the  editorial  team, 
who  interpret  it  to  mean  that  systems  engineers  are  interested  in  the  connection  between  systems  science  and 
systems  engineering. 

8.  Conclusion 

The  SEBoK  offers  the  student  of  systems  engineering  many  opportunities  for  data  mining.  This  article  has 
shown  some  preliminary  results  that  are  interesting.  They  include  the  most  heavily  cited  primary  references,  the 
areas  of  fewer  references,  the  longitudinal  distribution  of  references,  the  impact  of  various  institutions  on  the 
literature  of  systems  engineering,  and  the  extent  to  which  the  community  is  favoring  certain  topics  of  the  SEBoK. 
Additional  insights  will  be  gathered  as  the  SEBoK  continues  to  be  studied. 
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